Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task

VQA is an ambitious task aiming to answer any image-related question. However, in reality, it hard build such a system once for all since the needs of users are continuously updated, and has implement new functions. Thus, Continual Learning (CL) ability must developing advanced systems. Recently, pioneer work split dataset into disjoint sets study this topic. CL on involves not only expansion label (new Answer sets). It crucial how questions when deploying systems environments Visual scenes) requiring functions Question types). we propose CLOVE, benchmark On quEstion answering, which contains scene- function-incremental settings two aforementioned scenarios. In terms methodology, main difference between classification that former additionally expanding preventing forgetting reasoning mechanisms, while latter focusing class representation. real-data-free replay-based method tailored VQA, named Scene Graph as Prompt Symbolic Replay. Using piece scene graph prompt, replays pseudo graphs represent past images, along with correlated QA pairs. A unified model also proposed utilize current replayed data enhance its ability. Finally, experimental results reveal challenges CLOVE demonstrate effectiveness our method. Code available at https://github.com/showlab/CLVQA.